Functional stream processing workshop: Supercharge your streams

This workshop is a practical deep-dive into building reliable stream processing pipelines with Scala 3 and fs2.

Modern business applications are moving towards incremental processing. We design realtime data pipelines, event-driven architectures, reactive systems, and use stream-based I/O with databases, files and websockets. You may have heard of streams in media, but stream processing has many wider applications.

Just as functional programming has transformed the way we build applications, functional stream processing lets us write simple, safe, scalable incremental pipelines.

Target audience roles

This workshop is aimed at mid-level Scala developers and data engineers who want to develop practical skills in stream processing: understand what it is, how to use it and where to apply it. Afterwards, you might decide to design incremental pipelines at work with technologies such as Flink or Kafka, convert a batch-processing data pipeline into a realtime system, or use streams to simplify event processing logic.

Prerequisites

You'll benefit from having some functional Scala knowledge already (Scala 2 or 3), such as working with map and flatMap on lists. Absolutely no knowledge of stream processing or fs2 is needed.

Requirements

You’ll need a laptop with an IDE setup for basic Scala SBT projects.

Agenda / plan

Part 1: Introduction

• What is functional stream processing?: We’ll explore areas where stream processing is typically used, and compare it to conventional processing.
• Designing incremental pipelines: We’ll learn how to chain operations to compose larger pipelines.
• Reasoning about stream processing: We’ll learn to think of streams as iterative programs, develop a mental model of their execution, and use that to predict their output.
• Combining and splitting streams: We’ll explore the different methods of combining functional streams and explore what it means to split them.

Part 2: Handling side effects

• What are effect systems?: We’ll discuss the key properties of effect systems, and why they are important.
• Working with cats-effect IO: We’ll take a brief tour of cats-effect IO and its suspension of side effects.
• Working with effectful streams: We’ll see how IO lets us write many more practical stream programs.
• Error propagation, recovery and retries: We’ll learn how to recover from errors in an incremental process.
• Bracketing with streams: We’ll learn how to encode traditional try-catch-finally logic in an incremental process.
• Performance optimization: We’ll see how to detect performance problems in streams and address them with chunking.

Part 3: Concurrent systems

• Concurrency and parallelism: We’ll learn how to use streams to write safe, simple concurrent processes.
• Managing concurrent state: We’ll learn how to cleanly share state across concurrent streams.

Part 4: Streams at scale

• Scaling challenges in event-driven systems: We’ll explore the impact of load in incremental pipelines and the phenomena of backpressure.
• Designing simple, reliable systems with functional streams: We’ll analyse the differences between pull and push systems, and of functional streams and reactive systems, and see how functional streams greatly simplify system design.
• Handling backpressure with queues: We’ll use different types of queues to ensure a predictable behaviour under load.

Part 5: Application

We’ll use everything we’ve learned to create an interactive game. We’ll write an event-based stream for receiving user input, have a publisher-subscriber model for handling input and updating game state, and a separate concurrent render loop for displaying state to the screen. We’ll then enhance it with error handling, metrics, and logging, as we would any production-ready application.

Companies that use this technology

Apple, Disney, ITV, BBC, Xebia, SiriusXM, Jack Henry, JP Morgan, William Hill, and many more.